Acquiring and displaying images in real-time

ABSTRACT

An imaging device ( 100 ) for acquiring and displaying images in real-time, the imaging device comprising i) an imaging sensor ( 110 ) comprising a radiation sensitive array ( 120 ) for acquiring an image ( 142 ), ii) a readout circuit ( 140 ) connected to the radiation sensitive array for reading out the image, iii) a signal processor ( 160 ) for processing the image for obtaining a processed image ( 162 ), and iv) a display ( 180 ) for displaying the processed image, the radiation sensitive array being arranged in rows of sensor pixels and the display being arranged in rows of display pixels, and wherein the readout circuit is a rolling shutter circuit for sequentially reading out the rows of sensor pixels for sequentially providing subsets of pixels, the signal processor is configured for, on availability of one of the subsets of pixels, processing the subset of pixels for providing a processed subset of pixels, and the display is configured for, on availability of the processed subset of pixels, displaying the processed subset of pixels on a thereto corresponding subset of display pixels for displaying the processed image sequentially on the rows of display pixels.

This application is the U.S. national phase of International ApplicationNo. PCT/EP2011/052853 filed Feb. 25, 2011 which designated the U.S., theentire content of which is hereby incorporated by reference.

FIELD OF THE INVENTION

The invention relates to an imaging device for, and a method ofacquiring and displaying images in real-time. The invention furtherrelates to a helmet, a head mount, a rifle sight or a handheld devicecomprising the imaging device set forth.

Direct view systems are imaging devices in which images are acquired byan imaging component and then provided to a user in real-time by adisplay component. In such direct view systems, there may be a timedifference between the acquisition of an image and the display of theimage. This time difference is typically referred to as latency.Examples of direct view systems are night vision devices and telescopicviewing devices.

Latency is mostly undesirable, particularly when the direct view systemis intended for portable use. A reason for this is that the user may bemoving when using the direct view system. The latency may cause the userto perceive different motion through the display component than throughthe user's other senses of motion, e.g., the user's vestibular system.If the latency, and consequently the mismatch between the perceived andotherwise experienced motion is too high, the user may experience nauseaor motion sickness.

Many direct view systems are predominantly optical or opto-electronicalbased. For example, a night vision device may comprise optics and animage intensifier tube as imaging component and a phosphor screen asdisplay component. During operation, photons from a low light levelscene are converted into photoelectrons, multiplied by the imageintensifier tube and finally accelerated towards the phosphor screen fortheir conversion back into visible light. The latency of such a directview system is typically a few milliseconds as defined by the decay ofthe phosphorous screen, which is sufficiently low to avoid the userexperiencing motion sickness during portable use.

BACKGROUND OF THE INVENTION

It may be desirable to have a direct view system in which the image isintermediately available in digital form, i.e., being represented in adigital signal domain. This may allow the direct view system to employdigital signal processing to improve the image's quality, to overlayrelevant information onto the image, etc. Such a direct view system mayuse a semiconductor sensor for acquiring an image, a signal processorfor processing the image and an electronic display for displaying theimage.

SUMMARY OF THE INVENTION

The overall latency of such a digital signal domain based direct viewsystem may be relatively high. Disadvantageously, a user may experiencemotion sickness during portable use of such a direct view system.

It is an object of the invention to provide an imaging device for, and amethod of acquiring and displaying images in real-time with a reducedlatency, with the image being intermediately available in digital formfor allowing signal processing.

According to the invention, this object is realized in that imagingdevice is provided for acquiring and displaying images in real-time, theimaging device comprising i) an imaging sensor comprising a radiationsensitive array for acquiring an image, ii) a readout circuit connectedto the radiation sensitive array for reading out the image, iii) asignal processor for processing the image for obtaining a processedimage, and iv) a display for displaying the processed image, theradiation sensitive array being arranged in rows of sensor pixels andthe display being arranged in rows of display pixels, and wherein thereadout circuit is a rolling shutter circuit for sequentially readingout the rows of sensor pixels for sequentially providing subsets ofpixels, the signal processor is configured for, on availability of oneof the subsets of pixels, processing the subset of pixels for providinga processed subset of pixels, and the display is configured for, onavailability of the processed subset of pixels, displaying the processedsubset of pixels on a thereto corresponding subset of display pixels fordisplaying the processed image sequentially on the rows of displaypixels.

In a further aspect of the invention, a helmet, head mount, rifle sightor handheld device is provided comprising the imaging device set forth.

In a further aspect of the invention, a method is provided of acquiringand displaying images in real-time with an imaging device, the imagingdevice comprising i) an imaging sensor comprising a radiation sensitivearray for acquiring an image, ii) a readout circuit connected to theradiation sensitive array for reading out the image, iii) a signalprocessor for processing the image for obtaining a processed image, andiv) a display for displaying the processed image, the radiationsensitive array being arranged in rows of sensor pixels and the displaybeing arranged in rows of display pixels, and wherein the methodcomprises sequentially reading out the rows of sensor pixels with thereadout circuit for sequentially providing subsets of pixels, onavailability of one of the subsets of pixels, processing the subset ofpixels with the signal processor for providing a processed subset ofpixels, and on availability of the processed subset of pixels,displaying the processed subset of pixels with the display on a theretocorresponding subset of display pixels for displaying the processedimage sequentially on the rows of display pixels.

In a further aspect of the invention, a computer program is providedthat is stored on a computer-readable medium, the computer programcomprising instructions for causing a processor system to perform themethod set forth.

The measures according to the invention provide an imaging device foracquiring and displaying images in real-time, i.e., a direct viewsystem, and a method of operating the imaging device. Here, real-timerefers to a user using the imaging device to view images of a scene thatreflect the current scene as closely as possible in time. The imagingdevice comprises an imaging sensor, e.g., a semiconductor sensor such asa Complementary Metal-Oxide-Semiconductor (CMOS) sensor. The imagingsensor is used to convert radiation emitted or reflected from a sceneinto a digital representation of the scene, i.e., an image. For thatpurpose, the imaging sensor comprises a radiation sensitive array thatis arranged in rows of sensor pixels. The imaging device furthercomprises a readout circuit, i.e., a circuitry that reads out the imagefrom the radiation sensitive array by addressing and subsequentlyreading out the image from the rows of sensor pixels. The imaging devicefurther comprises a signal processor that is configured for processingthe image acquired from the imaging sensor using digital signalprocessing. As a result, a processed image is provided, that is thendisplayed on a display. The display is arranged in rows of displaypixels. As such, the rows of display pixels are used to display theimage that is acquired by the corresponding rows of sensor pixels of theradiation sensitive array.

The imaging device is configured for reducing a latency between theimage being acquired and the processed image being displayed. For thatpurpose, the readout circuit is a rolling shutter circuit. Rollingshutter is also known as line scan, and refers to a manner of readingout of the radiation sensitive array in which the rows of sensor pixelsare read out sequentially, i.e., one row after each other, or a subsetof rows after each other. As a consequence, the rows or subset of rowsthat have been read out correspond to different points in time. As aresult of the reading out of the rows of sensor pixels, the rollingshutter circuit sequentially provides subsets of pixels to the signalprocessor. In turn, the signal processor, upon receiving a subset ofpixels, processes the subset of pixels, and then provides a result ofthe processing, i.e., a processed subset of pixels, to the display. Eachprocessed subset of pixels is then displayed by the display on a theretocorresponding subset of display pixels. Thus, the rows of sensor pixelsare sequentially read out to provide a sequence of subsets of pixels,with a subset of pixels being processed after receipt by the signalprocessor and being displayed after receipt by the display.

It will be appreciated that the invention refers to reading, processingand displaying of rows of pixels. A reason for this is that sensors anddisplays are commonly read-out or written-to in a row-based manner.However, it will be appreciated that the present invention is equallyapplicable to the reading, processing and displaying of columns ofpixels, e.g., a reading circuit may be arranged for sequentially readingout columns of sensor pixels.

The invention is partially based on the recognition that in traditionaldirect view systems, a so-termed snapshot mode is used for reading outthe image from the radiation sensitive array. Here, the entire image isfirst read out and stored in a frame buffer memory, and only when theentire image has been stored, the image is processed and subsequentlydisplayed in its entirety. Disadvantageously, the latency introduced bystoring the image in a frame buffer before or during the processing maycause a user to experience motion sickness during portable use of thedirect view system.

The effect of the aforementioned measures is that the imaging device isconfigured for displaying a subset of pixels on a subset of displaypixels as soon as possible after the corresponding subset of sensorpixels has been read out. Thus, the imaging device is configured fordirectly providing each portion of the image that has been read out,i.e., each subset of pixels, to the signal processor and subsequentlyeach processed portion to the display. As a result, the overall latencyof the imaging device is reduced. Advantageously, the latency of theimaging device is sufficiently reduced for avoiding a user experiencingmotion sickness during portable use of the imaging device.

Optionally, the imaging device comprises a further imaging sensor and afurther readout circuit, the further imaging sensor comprising a furtherradiation sensitive array for acquiring a further image, the furtherreadout circuit being connected to the further radiation sensitive arrayfor reading out the further image, the further radiation sensitive arraybeing arranged in rows of further sensor pixels, the further readoutcircuit being a further rolling shutter circuit for sequentially readingout the rows of further sensor pixels for sequentially providing furthersubsets of pixels, and wherein the imaging device is configured forsynchronously displaying the image and the further image on the displayby the rolling shutter circuit and the further rolling shutter circuitbeing configured for synchronously providing the subset of pixels andone of the further subsets of pixels by substantially synchronouslyreading out corresponding portions of the image and the further image,and the signal processor being configured for, on availability of thesubset of pixels and the further subset of pixels, combining the subsetof pixels with the further subset of pixels for obtaining the processedsubset of pixels.

The imaging device is configured for synchronously displaying the imageand a further image on the display. For acquiring the further image, theimaging device comprises a further imaging sensor. In order tosynchronously provide corresponding portions of both images to thesignal processor for processing and subsequent display, the rollingshutter circuit and the further rolling shutter circuit are configuredfor substantially synchronously reading out the corresponding portionsof the image and the further image. As a result, the subset of pixelsand the further subset of pixels are provided synchronously.

The effect of the aforementioned measures is that the imaging device isconfigured for acquiring and displaying two images simultaneously whileat the same time also reducing the latency between said acquiring anddisplaying. Advantageously, fewer buffer memories are needed in theimaging device, as there is less or no need for compensating for amismatch in a timing of portions of the image and portions of thefurther image becoming available for subsequent processing and display.Consequently, the cost and/or the complexity of the imaging device isreduced.

Optionally, the imaging sensor is a visible light imaging sensor forsensing visible light and the further imaging sensor is a thermalimaging sensor for sensing infrared radiation for enabling synchronouslydisplaying a visible light image and a thermal image on the display.Advantageously, the imaging device simultaneously acquires and displaysthe visible light and the thermal radiation of a scene while providing areduced latency.

Optionally, the signal processor is configured for combining the subsetof pixels with the further subset of pixels by fusing the subset ofpixels with the further subset of pixels for obtaining as the processedimage an image fusion of the image with the further image. Image fusionoffers an intuitive way of combining two images, and in particular, tworelated images of a same scene. Advantageously, the thermal radiation ofa scene may be visualized as colours overlaid over the visible light ofa scene for offering an intuitive way of displaying visible light andthermal radiation of a scene to a user.

Optionally, the radiation sensitive array has a first spatialresolution, the further radiation sensitive array has a second spatialresolution, the second spatial resolution being lower than the firstspatial resolution and the further rolling shutter circuit beingconfigured for reading out the further image with a second readout speedthat is lower than a first readout speed of the rolling shutter circuitfor enabling said synchronously providing the subset of pixels and thefurther subset of pixels.

The further rolling shutter circuit thus uses a lower readout speed toensure the synchronously providing of the corresponding portions of theimage and the further image to the signal processor. Adapting thereadout speed is an efficient way of compensating for said difference inspatial resolutions, as, e.g., a same readout speed would typicallyrequire frequent intermediate pausing of the reading out to ensure theaforementioned synchronicity. Advantageously, a lower readout speedresults in a lower power consumption of the further rolling shuttercircuit, and consequently, of the imaging device.

Optionally, the rolling shutter circuit is configured for reading outthe image with the first readout speed within an imaging frame time, andthe further rolling shutter circuit is configured for reading out thefurther image with the second readout speed within the imaging frametime. The readout speeds are thus adapted to read the image and thefurther image within the same imaging frame time. As a consequence, aratio of the first readout speed to the second readout speed equals theratio of the first spatial resolution and the second spatial resolution.Advantageously, no intermediate pausing of the reading out is needed toensure the aforementioned synchronicity.

Optionally, the rolling shutter circuit is clocked at a first pixelclock for providing the first readout speed and the further rollingshutter circuit is clocked at a second pixel clock for providing thesecond readout speed. The pixel clock of each rolling shutter circuit isthus adapted to the needed readout speed. Advantageously, the lowerpixel clock of the further rolling shutter circuit results in a lowerpower consumption.

Optionally, the imaging device comprises a scaler for spatially scalingthe further subset of pixels for providing as the further image a scaledimage having the first spatial resolution. A scaler provides anefficient way of adjusting the spatial resolution of the further imageto the spatial resolution of the image. Advantageously, the furtherrolling shutter circuit may not need to compensate for the difference inspatial resolution by providing the further subset of pixels to abuffer, and the signal processor repeatedly reading the same furthersubset of pixels from the buffer. Advantageously, the further image isdisplayed with a better image quality, and in particular, with a betterspatial definition of edges. Advantageously, the further image may beoverlaid on the image, with overlaid portions of both images beingassociated with a same portion of a scene.

Optionally, the scaler is configured for performing the spatial scalingusing at least one technique out of the group of: pixel repetition,first order linear interpolation, higher order linear interpolation andnon-linear interpolation techniques. The aforementioned interpolationtechniques are particularly well-suited for spatial scaling.

Optionally, the signal processor comprises an image processing pipelinefor obtaining a pipelined processing of the subsets of pixels.Performing the image processing in a pipelined manner, as is known fromthe technical field of processor design and architecture, provides aprocessing higher throughput. Advantageously, the signal processor canaccept new subsets of pixels, or individual pixels of the new subset ofpixels, in each clock cycle. Advantageously, less buffering is needed tocope with the signal processor being unable to accept new pixels due tobeing occupied with processing of previous pixels.

Optionally, the rolling shutter circuit is configured for reading outthe image with a first readout speed, and wherein the imaging device isconfigured for establishing the first readout speed in dependence on anamount of radiation impinging on the radiation sensitive array. Byestablishing the first readout speed in dependence on an amount ofradiation impinging on the radiation sensitive device, a trade-off canbe established between a needed exposure time of the radiation sensitivearray, and a difference in acquisition between a top portion of theimage and a bottom portion, which results in so-termed skew effects.Advantageously, the first readout speed can be increased if sufficientradiation is impinging on the radiation sensitive array, therebyreducing said skew effects.

Optionally, the imaging device comprises an image intensifier forproviding intensified visible light to the imaging sensor. By using animage intensifier, the imaging sensor is able to acquire an image in lowlight conditions with an improved signal-to-noise ratio. An imagingdevice comprising an image intensifier is typically also known as anight vision device or low light level image intensifier.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects of the invention are apparent from and will beelucidated with reference to the embodiments described hereinafter. Inthe drawings,

FIG. 1 shows a timing diagram of a direct view system;

FIG. 2 shows an imaging device comprising an imaging sensor;

FIG. 3 shows the imaging sensor and a display;

FIG. 4 shows a timing diagram of the imaging device;

FIG. 5 shows an alternate representation of the timing diagram of FIG.4;

FIG. 6 shows an imaging device comprising a further imaging sensor;

FIG. 7 shows an imaging sensor and the further imaging sensor;

FIG. 8 shows a timing diagram of the imaging device;

FIG. 9 shows an imaging device comprising a scaler;

FIG. 10 shows a schematic functioning of a readout circuit;

FIG. 11 shows a method for acquiring and displaying images in real-time;

FIG. 12 shows a computer program stored on a computer-readable medium.

DETAILED DESCRIPTION OF EMBODIMENTS

FIG. 1 shows a timing diagram of a direct view system comprising a framebuffer memory. Here, the horizontal axis is indicative of time, whereasthe vertical axis is used for visually differentiating between timingsof a reading of an image, a processing of the image and a displaying ofthe image. Here, SS₁ indicates a time period for reading an image froman imaging sensor. The reading SS₁ comprises storing the read image in aframe buffer memory. In such a direct view system, processing takes onlyplace when the image is entirely stored in the frame buffer memory.Thus, after the reading SS₁ has completed, the direct view systemcommences processing PP₁ the image. Typically, portions of the image areread out from the frame buffer memory, processed, and then written backto the frame buffer memory or to a further frame buffer memory. Finally,after the processing PP₁ has completed, the direct view system commencesdisplaying DD₁ the image. Consequently, a time period between a startTT₁ of the reading SS₁ of the image and a start TT₂ of the displayingDD₁ of the image indicates a minimum delay, or latency LL, that a userexperiences between an change in a scene and the displayed imagereflecting said change.

For increasing a throughput of the direct view system, the reading,processing and displaying may be pipelined. This means that while theprocessing PP₁ of the image takes place, a reading SS₂ of a followingimage may take place. Similarly, while the displaying DD₁ of the imagetakes place, a processing PP2 of the following image may take place,etc. It is noted that such pipelining increases a throughput of thedirect view system, i.e., allows a system to read, process and displaymore images in a given time period. However, said pipelining does notaffect the latency LL of the direct view system.

FIG. 2 shows an imaging device 100 for acquiring and displaying imagesin real-time. The imaging device 100 comprises an imaging sensor 110,and the imaging sensor 110 comprises a radiation sensitive array 120 foracquiring an image 142. The imaging device 100 further comprises areadout circuit 140 connected to the radiation sensitive array 120 forreading out the image 142, and a signal processor 160 for processing theimage 142 for obtaining a processed image 162. For that purpose, thereadout circuit 140 is shown to be connected to the signal processor160. The imaging device 100 further comprises a display 180 for finallydisplaying the processed image 162 on the display 180.

FIG. 3 shows the imaging sensor 110 comprising the radiation sensitivearray 120. Also shown is that the radiation sensitive array 120 isarranged in rows of sensor pixels 122. It is noted that, although notexplicitly indicated in FIG. 3, the radiation sensitive array 120 isalso arranged in columns of sensor pixels as a consequence of being anarray. Also shown in FIG. 3 is the display 180 as a display pixel arraywhich is arranged in rows of display pixels 182. It is noted that,although not explicitly indicated in FIG. 3, the display 180 is alsoarranged in columns of display pixels as a consequence of being anarray.

During operation of the imaging device 100, the readout circuit 140sequentially reads out the rows of sensor pixels 122 for sequentiallyproviding a subset of pixels 124. This reading is indicated in thetiming diagrams shown in FIGS. 4 and 5. Here, S₁ indicates a time periodfor reading the image 142 from the imaging sensor 110. FIG. 4 shows thetime period in a similar manner as FIG. 1 for allowing a comparison withthe aforementioned direct view system. In FIG. 5, the horizontal axis isindicative of time, whereas the vertical axis is indicative of a rownumber, with R_(n) indicating a top row of the radiation sensitive array120 and R₀ indicating a bottom row. Thus, FIG. 5 shows the readoutcircuit 140 reading out the row R_(n) at the beginning of the timeperiod S₁ and the row R₀ at its end. Consequently, during the readingS₁, all rows R_(n) to R₀ are read out sequentially.

The above described reading S₁ of the radiation sensitive array 120 isachieved by the readout circuit 140 being a rolling shutter circuit. Therolling shutter circuit 140 differs from a circuit configured forreading out the radiation sensitive array 120 using a snapshot shutter.The basic operating principle of a rolling shutter circuit is that theradiation sensitive array 120 is addressed in a row-by-row, i.e., on aline-by-line basis for i) initializing an exposure of a row and ii)reading out its contents after said exposure. This is typically achievedby the use of two pointers, each addressing respective rows in theradiation sensitive array 120. One pointer provides a reset of acurrently addressed row for initializing an exposure of the row, whereasthe other pointer addresses a row that is to be read out. The differencein location between the two pointers is the effective exposure time,i.e., if the ‘reset’ pointer trails the ‘readout’ pointer by only onerow, the exposure time is maximized.

In contrast, a circuit that uses a snapshot shutter typically exposesthe entire radiation sensitive array simultaneously before reading outthe entire image from the radiation sensitive array into a frame buffermemory. During the time needed for reading out the entire image, theradiation sensitive array is not configured for exposure anymore.Disadvantageously, the exposure time provided by said circuit is lessthan that of a rolling shutter circuit. A shorter exposure timetypically results in an image that has a worse signal-to-noise ratio,i.e., is noisier. A publication “EBAPS: Next Generation, Low Power,Digital Night Vision”, Aebi et al., Intevac Corporation, OPTRO 2005symposium, Paris, France, describes using a rolling shutter circuit formaximizing the exposure time of a camera sensor.

The pointers of the rolling shutter circuit 140 may be increased by aninternal state machine in the imaging sensor 110 itself, i.e., therolling shutter circuit 140 may be part of the imaging sensor 110.External logic, e.g., a Field Programmable Gate Array (FPGA) locatedoutside of the imaging sensor 110 may be used to clock the state machineand to program the distance between the two pointers for determining theexposure time.

By sequentially reading out the rows of sensor pixels 122, the rollingshutter circuit 140 sequentially provides a subset of pixels 124. Thesubset of pixels 124 may comprise the pixels of an entire row, or of asubset of rows. The subset of pixels may also comprise a subset ofpixels from a single row, e.g., a single pixel or multiple neighbouringpixels. The rolling shutter circuit 140 provides the subset of pixels124 to the signal processor 160, which, on availability of the subset ofpixels 124, processes the subset of pixels to provide a processed subsetof pixels. By processing the sequentially provided subset of pixels, thesignal processor 160 effectively processes the image 142 and provides aprocessed image 162. This processing is indicated in the timing diagramsshown in FIGS. 4 and 5. Here, P₁ indicates a time period for processingthe image 142. FIG. 5 shows the readout circuit 140 processing the rowR_(n) at the beginning of the time period P₁ and the row R₀ at its end.Consequently, during the processing P₁, all rows R_(n) to R₀ areprocessed sequentially.

The time delay between the reading S₁ and the processing P₁, as shown inFIGS. 4 and 5, is dependent on, amongst others, a size of the subset ofpixel 124. For example, if the subset of pixels 124 comprises the pixelsof a row of sensor pixels 122, the processing P₁ is delayed with respectto the reading S₁ by at least a time period needed for reading andproviding said subset of pixels 124 to the signal processor 160. It willbe appreciated, however, that said time delay is significantly less thana time delay corresponding to an entire reading out of the image 142 dueto the sequential providing of the subset of pixels 124.

On availability of the processed subset of pixels, the display 180displays the processed subset of pixels on a thereto correspondingsubset of display pixels 184. By displaying the sequentially providedprocessed subset of pixels, the display 180 effectively displays theprocessed image 162. This displaying is indicated in the timing diagramsshown in FIGS. 4 and 5. Here, D₁ indicates a time period for displayingthe processed image 162. FIG. 5 shows the display 180 displaying the rowR_(n) at the beginning of the time period D₁ and the row R₀ at its end.It will be appreciated that for the time delay between the displaying D₁and the processing P₁, similar considerations holds as for the timedelay between the reading S₁ and the processing P₁. The imaging deviceshown in FIG. 2 thus provides a latency L that corresponds to a timedelay between the reading S₁ and the displaying D₁. Moreover, it will beappreciated that a reading S₂ of a following image may commence afterthe reading S₁ of the image has finished. Similarly, a processing P₂ maycommence after the processing P₁ has finished, and a displaying D₂ maycommence after the displaying D₁ has finished.

FIG. 6 shows an imaging device 200. The imaging device comprises, nextto the imaging sensor 110 and the readout circuit 140, also a furtherimaging sensor 210 and a further readout circuit 240. The furtherimaging sensor 210 comprises a further radiation sensitive array 220 foracquiring a further image 242. For reading out the further image 242,the further readout circuit 240 is connected to the further radiationsensitive array 220.

FIG. 7 shows the further imaging sensor 210 comprising the furtherradiation sensitive array 220 next to the aforementioned imaging sensor110 and radiation sensitive array 110. Also shown is that the furtherradiation sensitive array 220 is arranged in rows of further sensorpixels 222. It is noted that, although not explicitly indicated in FIG.7, the further radiation sensitive array 220 is also arranged in columnsof sensor pixels as a consequence of being an array. FIG. 7 also showsthe radiation sensitive array 120 having a first spatial resolution, thefurther radiation sensitive array 220 having a second spatialresolution, with the second spatial resolution being lower than thefirst spatial resolution. As a consequence, the radiation sensitivearray 120 is build up by n+1 rows, i.e., row R₀ to row R_(n), whereasthe radiation sensitive array 220 is build up by m+1 rows, i.e., row R₀to row R_(n), with m being smaller than n. This configuration is assumedin the remainder of the description of the imaging device of FIG. 6.However, it is noted that the second spatial resolution may also beequal to or larger than the first spatial resolution. Moreover, it isnoted that spatial resolution may refer to an image's horizontal orvertical resolution, e.g., having 1280 pixels or 1024 lines, or to acombined resolution, e.g., a 1.3 megapixel image.

Referring to FIG. 6 again, the readout circuit 140 is configured as afurther rolling shutter circuit for sequentially reading out the rows offurther sensor pixels 222. During operation of the imaging device 200,the rolling shutter circuit 140 and the further rolling shutter circuit240 synchronously provide the subset of pixels 124 and the furthersubset of pixels 224 by substantially synchronously reading outcorresponding portions of the image 142 and the further image 242. Here,corresponding portions refer to portions of the image that have anassociated image contents. For example, when the imaging sensor 110 is avisible light imaging sensor for sensing visible light 112 and thefurther imaging sensor 210 is a thermal imaging sensor for sensinginfrared radiation 212, the image 142 may be a visible light image of ascene and further image 242 may be a thermal image of the same scene.Consequently, corresponding portions may refer to, e.g., the top rowR_(n) of the image 142 corresponding to the top row R_(m) of the furtherimage 242, the bottom row R₀ of the image 142 corresponding to a samebottom row R₀ of the further image 242, etc. This enables the signalprocessor 260 to, on availability of the subset of pixels 124 and thefurther subset of pixels 224, combine both subsets of pixels to obtainthe processed subset of pixels for, e.g., providing a processed image262 in which the thermal image is overlaid on top of the visible lightimage. It is noted that corresponding portions may also refer to, e.g.,when the imaging sensor 110 acquires a left-hand view and the furtherimaging sensor 210 acquires a right-hand view, portions that have a samevertical position in either image.

In order to compensate for the second spatial resolution being lowerthan the first spatial resolution, the further rolling shutter circuit240 is configured for reading out the further image 242 with a secondreadout speed that is lower than a first readout speed of the rollingshutter circuit 140. This is shown in FIG. 8, where a similar timingdiagram is shown as in FIG. 5, with additionally a time period of areading I_(i) of the further image 242 being indicated. It will beappreciated that, with the number of rows R_(m) of the further image 242being lower than the number of rows R_(n) of the image 142, the secondreadout speed needs to be lower to enable the reading I₁ of the furtherimage 242 within a same time interval as the reading S₁ of the image142. This is reflected in a lower slope of the reading I₁ with respectto a horizontal axis when compared to the reading S₁.

FIG. 8 shows the first readout speed being selected for reading S₁ theimage 142 within an imaging frame time T_(i) and the second readoutspeed being selected for reading I₁ the further image 242 within thesame imaging frame time T_(i). The imaging frame time T_(i) is directlycoupled to the imaging frame rate, e.g., is 1/60 s=0.0167 ms with a 60Hz imaging frame rate. For maximizing an exposure of the radiationsensitive array 120, the first readout speed is selected for reading S₁the image 142 in substantially said imaging frame time T_(i).Furthermore, the second readout speed is selected for substantiallyreading I₁ the further image 242 within the same imaging frame timeT_(i). It will be appreciated that the resulting ratio between the firstreadout speed and the second readout speed inherently follows from theaforementioned configuration of the imaging device 200 for synchronouslyreading out corresponding portions of the image 142 and the furtherimage 242.

Moreover, in the example depicted in FIG. 8, the first readout speed andthe second readout speed are selected for providing a reading S₁ of theimage 142 and a reading I₁ of the further image 242 that covers theentire imaging frame time T_(i). This may follow out of a preference forthe aforementioned maximizing of an exposure time of the radiationsensitive array 120 and/or of the further radiation sensitive array 220.However, the reading S₁ and the reading I₁ may also be faster, e.g.,being completed before an end of the imaging frame time T_(i). Thisreduces the time difference between the reading S₁ of the top and bottompart of the image 142, and thus may reduce or avoid so-termed skewartefacts in the image 142. These artefacts are known to occur when saidtime difference is relatively large. Also, the imaging device 200 may beconfigured for establishing the first readout speed in dependence on anamount of radiation 112 impinging on the radiation sensitive array 120.As such, the imaging device 200 may dynamically determine a compromisebetween a needed exposure time and the aforementioned skew effects.

The rolling shutter circuit 140 may be clocked at a first pixel clockfor providing the first readout speed and the further rolling shuttercircuit 240 may be clocked at a second pixel clock for providing thesecond readout speed. Since the second readout speed is lower than thefirst readout speed, the second pixel clock is also lower than the firstpixel clock. For example, when the second spatial resolution of thefurther image 242 is horizontally and vertically one fourth of that ofthe first spatial resolution of the image 142, e.g., 320 by 256 pixelswith respect to 1280 by 1024 pixels, the first readout circuit 140 maybe clocked at a system clock of, e.g., 44 MHz, whereas the secondreadout circuit 240 may be clocked at one sixteenth of that systemclock, i.e., a 2.75 MHz pixel clock. Since a lower clock rate typicallyresults in lower power consumption, the power consumption of the imagingdevice 200 may be reduced. Alternatively, the second readout circuit 240may be clocked at 44 MHz as well, but may be configured to, on average,only provide one pixel every sixteenth clock cycle.

Since the second spatial resolution is lower than the first spatialresolution, it may be needed to scale the further image 242 to the firstspatial resolution or to a spatial resolution of the display 180. It isnoted that this may not be needed in all cases, e.g., when the furtherimage 242 is inserted as a so-termed Picture-in-Picture (PiP) into theimage 142. FIG. 9 shows an imaging device 300 comprising a scaler 250for providing as the further image 242 a scaled image 252 having thefirst spatial resolution. Here, the further rolling shutter circuit 240is shown to be connected to the scaler 250 for providing the furtherimage 242 to the scaler 250, and the scaler 250 is shown to be connectedto the signal processor 260 for providing the scaled image 252 to thesignal processor 260. The scaler 250 may comprise line buffer memoriesfor enabling spatial scaling in a vertical direction. The spatialscaling may comprise performing a zero order linear interpolationtechnique, i.e., a so-termed pixel repetition or nearest neighbourinterpolation, as is known from the technical field of image processing.The spatial scaling may also comprise techniques such as first orderlinear interpolation, e.g., bilinear interpolation, higher order linearinterpolation and non-linear interpolation techniques. Such techniquestypically introduce fewer interpolation artefacts.

It is noted that the imaging device 300 does not need to comprise anexplicit scaler 250. Instead, a buffer may be used that effectivelyfunctions as a scaler. For example, the further rolling shutter circuit240 may comprise a so-termed First-In-First-Out (FIFO) buffer, as isknown from the technical field of processor design and architectures.The further rolling shutter circuit 240 may then read a row R_(m), as isshown schematically in FIG. 10, from the further radiation sensitivearray 220. The reading may be performed using a 2.75 MHz pixel clock.The read out row R_(m) may then be buffered in the FIFO, and read outwith a higher pixel clock, e.g., a 44 MHz pixel clock, for providing therow R_(m) repeatedly at a same readout speed as the row R_(n) that isread out by the rolling shutter circuit 140. It is noted that such useof a FIFO buffer effectively performs a nearest neighbour interpolation,although it may conventionally not considered being a scaler. Also, itwill be appreciated that such functionality may be also implemented inthe signal processor 260 itself, i.e., the signal processor 260 maycomprise the FIFO for performing said buffering.

FIG. 11 shows a method 300 of acquiring and displaying images inreal-time with an imaging device, the imaging device comprising i) animaging sensor comprising a radiation sensitive array for acquiring animage, ii) a readout circuit connected to the radiation sensitive arrayfor reading out the image, iii) a signal processor for processing theimage for obtaining a processed image, and iv) a display for displayingthe processed image, the radiation sensitive array being arranged inrows of sensor pixels and the display being arranged in rows of displaypixels, and wherein the method comprises sequentially reading 340 outthe rows of sensor pixels with the readout circuit for sequentiallyproviding a subset of pixels, on availability of the subset of pixels,processing 360 the subset of pixels with the signal processor forproviding a processed subset of pixels, and on availability of theprocessed subset of pixels, displaying 380 the processed subset ofpixels with the display on a thereto corresponding subset of displaypixels for displaying the processed image sequentially on the rows ofdisplay pixels.

FIG. 12 shows a computer readable medium 400 comprising a computerprogram 420, the computer program 420 comprising instructions forcausing a processor system to perform the method 300 as shown in FIG.11. The computer program 420 may be embodied on the computer readablemedium 400 as physical marks or by means of magnetization of thecomputer readable medium 400. However, any other suitable embodiment isconceivable as well. Furthermore, it will be appreciated that, althoughthe computer readable medium 400 is shown in FIG. 12 as an optical disc,the computer readable medium 400 may be any suitable computer readablemedium, such as a read-only-memory or random-access memory, e.g., solidstate memory, flash memory, etc.

It will be appreciated that the present invention may be used forvarious kinds of imaging sensors, and thus is not limited to, e.g.,visible light imaging sensors or thermal sensors. Moreover, combiningthe image 142 with the further image 242 may comprise fusing the image142 with the further image 242 by, e.g., overlaying certain elements ofthe further image 242 on top of the image 142. However, the combiningmay also comprise creating a processed image 262 that comprises aside-by-side, picture-in-picture or similar spatial arrangement of theimage 142 and the further image 242.

The signal processor 160 may employ various kinds of signal processingnext to the aforementioned combining of fusing of the image 142 and thefurther image 242. For example, the signal processor 160 may performvarious kinds of image processing, as are known from the technical fieldof image processing, such as non-uniformity correction, histogramequalization, noise reduction, sharpening, colour mapping, etc.Moreover, to increase a throughput of the signal processor 160, thesignal processing may comprise an image processing pipeline forobtaining a pipelined processing of the subset of pixels 124.

The display 180 may be a micro Organic Light Emitting Diode (OLED) orLiquid Crystal (LC) based display. The imaging sensor 110 may be a CMOSsensor. The signal processor 160 may be embodied in a FPGA. The imagingsensor 160 may be configured for providing synchronization information,e.g., so-termed horizontal and vertical SYNC signals. These may be usedby the imaging device 100 to synchronize the reading, the processing andthe displaying of the image 142. The synchronization information mayalso be used for synchronizing reading the further image 242 using thefurther readout circuit 240.

It will be appreciated that the above description for clarity hasdescribed embodiments of the invention with reference to differentfunctional units. However, it will be apparent that any suitabledistribution of functionality between different functional units orprocessors may be used without detracting from the invention. Forexample, functionality illustrated to be performed by separateprocessors or controllers may be performed by the same processor orcontrollers. Hence, references to specific functional units are only tobe seen as references to suitable means for providing the describedfunctionality rather than indicative of a strict logical or physicalstructure or organization.

The invention can be implemented in any suitable form includinghardware, software, firmware or any combination of these. The inventionmay optionally be implemented at least partly as computer softwarerunning on one or more data processors and/or digital signal processors.The elements and components of an embodiment of the invention may bephysically, functionally and logically implemented in any suitable way.Indeed the functionality may be implemented in a single unit, in aplurality of units or as part of other functional units. As such, theinvention may be implemented in a single unit or may be physically andfunctionally distributed between different units and processors.

Although the present invention has been described in connection withsome embodiments, it is not intended to be limited to the specific formset forth herein. Rather, the scope of the present invention is limitedonly by the accompanying claims. Additionally, although a feature mayappear to be described in connection with particular embodiments, oneskilled in the art would recognize that various features of thedescribed embodiments may be combined in accordance with the invention.In the claims, the term comprising does not exclude the presence ofother elements or steps.

Furthermore, although individually listed, a plurality of means,elements or method steps may be implemented by e.g. a single unit orprocessor. Additionally, although individual features may be included indifferent claims, these may possibly be advantageously combined, and theinclusion in different claims does not imply that a combination offeatures is not feasible and/or advantageous. Also the inclusion of afeature in one category of claims does not imply a limitation to thiscategory but rather indicates that the feature is equally applicable toother claim categories as appropriate. Furthermore, the order offeatures in the claims do not imply any specific order in which thefeatures must be worked and in particular the order of individual stepsin a method claim does not imply that the steps must be performed inthis order. Rather, the steps may be performed in any suitable order. Inaddition, singular references do not exclude a plurality. Thusreferences to “a”, “an”, “first”, “second” etc do not preclude aplurality. Reference signs in the claims are provided merely as aclarifying example shall not be construed as limiting the scope of theclaims in any way.

The invention claimed is:
 1. A night vision device arranged foracquiring and displaying images in real-time, the night vision devicecomprising: i) an imaging sensor comprising a radiation sensitive arrayfor acquiring an image, ii) a readout circuit connected to the radiationsensitive array for reading out the image, iii) a signal processor forprocessing the image for obtaining a processed image, and iv) a displayfor displaying the processed image, the radiation sensitive array beingarranged in rows of sensor pixels and the display being arranged in rowsof display pixels, and wherein: the readout circuit is a rolling shuttercircuit for sequentially reading out the rows of sensor pixels by a)initializing an exposure of a row, and b) reading out its contents aftersaid exposure, for sequentially providing subsets of pixels), eachsubset of pixels corresponding to one row or a subset of the rows ofsensor pixels or a subset of pixels from a single row; the signalprocessor is configured for, on availability of one of the subsets ofpixels, processing the subset of pixels for providing a processed subsetof pixels; and the display is configured for, on availability of theprocessed subset of pixels, displaying the processed subset of pixels ona thereto corresponding subset of display pixels for displaying theprocessed image sequentially on the rows of display pixels, and whereinthe direct view system comprises a further imaging sensor and a furtherreadout circuit, the further imaging sensor comprising a furtherradiation sensitive array for acquiring a further image, the furtherreadout circuit being connected to the further radiation sensitive arrayfor reading out the further image, the further radiation sensitive arraybeing arranged in rows of further sensor pixels, the further readoutcircuit being a further rolling shutter circuit for sequentially readingout the rows of further sensor pixels for sequentially providing furthersubsets of pixels, and wherein the direct view system is configured forsynchronously displaying the image and the further image on the displayby: (1) the rolling shutter circuit and the further rolling shuttercircuit being configured for synchronously providing the subset ofpixels and one of the further subsets of pixels by substantiallysynchronously reading out corresponding portions of the image and thefurther image; and (2) the signal processor being configured for, onavailability of the subset of pixels and the further subset of pixels,combining the subset of pixels with the further subset of pixels forobtaining the processed subset of pixels, wherein the radiationsensitive array has a first spatial resolution, the further radiationsensitive array has a second spatial resolution, the second spatialresolution being lower than the first spatial resolution and the furtherrolling shutter circuit being configured for reading out the furtherimage with a second readout speed that is lower than a first readoutspeed of the rolling shutter circuit for enabling said synchronouslyproviding the subset of pixels and the further subset of pixels, andwherein the rolling shutter circuit is clocked at a first pixel clockfor providing the first readout speed and the further rolling shuttercircuit is clocked at a second pixel clock for providing the secondreadout speed.
 2. The night vision device according to claim 1, whereinthe imaging sensor is a visible light imaging sensor for sensing visiblelight and the further imaging sensor is a thermal imaging sensor forsensing infrared radiation for enabling synchronously displaying avisible light image and a thermal image on the display.
 3. The nightvision device according to claim 1, wherein the signal processor isconfigured for combining the subset of pixels with the further subset ofpixels by fusing the subset of pixels with the further subset of pixelsfor obtaining as the processed image an image fusion of the image withthe further image.
 4. The night vision device according to claim 1,wherein the rolling shutter circuit is configured for reading out theimage with the first readout speed within an imaging frame time (Ti),and the further rolling shutter circuit is configured for reading outthe further image with the second readout speed within the imaging frametime.
 5. The night vision device according to claim 1, wherein the nightvision device comprises a scaler for spatially scaling the furthersubset of pixels for providing as the further image a scaled imagehaving the first spatial resolution.
 6. The night vision deviceaccording to claim 5, wherein the scaler is configured for performingthe spatial scaling using at least one technique out of the group of:pixel repetition, first order linear interpolation, higher order linearinterpolation and non-linear interpolation techniques.
 7. The nightvision device according to claim 1, wherein the signal processorcomprises an image processing pipeline for obtaining a pipelinedprocessing of the subsets of pixels.
 8. The night vision deviceaccording to claim 1, wherein the rolling shutter circuit is configuredfor reading out the image with a first readout speed, and wherein thenight vision device is configured for establishing the first readoutspeed in dependence on an amount of radiation impinging on the radiationsensitive array.
 9. The night vision device according to claim 1,wherein the night vision device comprises an image intensifier forproviding intensified visible light to the imaging sensor.
 10. A helmet,head mount, rifle sight or handheld device comprising the night visiondevice according to claim
 1. 11. A method of acquiring and displayingimages in real-time with a night vision device, the night vision devicecomprising i) an imaging sensor comprising a radiation sensitive arrayfor acquiring an image, ii) a readout circuit connected to the radiationsensitive array for reading out the image, iii) a signal processor forprocessing the image for obtaining a processed image, and iv) a displayfor displaying the processed image, the radiation sensitive array beingarranged in rows of sensor pixels and the display being arranged in rowsof display pixels, and wherein the method comprises: sequentiallyreading out the rows of sensor pixels with the readout circuit by (a)initializing an exposure of a row, and (b) reading out its contentsafter said exposure, for sequentially providing subsets of pixels, eachsubset of pixels corresponding to one row or a subset of the rows ofsensor pixels or a subset of pixels from a single row; on availabilityof one of the subsets of pixels, processing the subset of pixels withthe signal processor for providing a processed subset of pixels; and onavailability of the processed subset of pixels, displaying the processedsubset of pixels with the display on a thereto corresponding subset ofdisplay pixels for displaying the processed image sequentially on therows of display pixels, wherein the night vision device comprises afurther imaging sensor and a further readout circuit, the furtherimaging sensor comprising a further radiation sensitive array foracquiring a further image, the further readout circuit being connectedto the further radiation sensitive array for reading out the furtherimage, the further radiation sensitive array being arranged in rows offurther sensor pixels, the further readout circuit being a furtherrolling shutter circuit for sequentially reading out the rows of furthersensor pixels for sequentially providing further subsets of pixels, andwherein the method comprises synchronously displaying the image and thefurther image on the display by: synchronously providing the subset ofpixels and one of the further subsets of pixels by substantiallysynchronously reading out corresponding portions of the image and thefurther image; and on availability of the subset of pixels and thefurther subset of pixels, combining the subset of pixels with thefurther subset of pixels for obtaining the processed subset of pixels,wherein the radiation sensitive array has a first spatial resolution,the further radiation sensitive array has a second spatial resolution,the second spatial resolution being lower than the first spatialresolution and the method further comprising reading out the furtherimage with a second readout speed that is lower than a first readoutspeed of the rolling shutter circuit for enabling said synchronouslyproviding the subset of pixels and the further subset of pixels, whereinmethod comprises clocking the rolling shutter circuit at a first pixelclock for providing the first readout speed and clocking the furtherrolling shutter circuit at a second pixel clock for providing the secondreadout speed.
 12. A computer program stored on a non-transitorycomputer-readable medium, the computer program comprising instructionsfor causing a processor system to perform the method according to claim11.